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Executive Summary 


This study examines how district leaders use research in their instructional decision-making in 
four large urban districts. School district central offices make consequential decisions about 
teaching and learning every day that impact the educational opportunities and outcomes for 
millions of students in our nation’s public schools. Given the consequential nature of these 
decisions, it is important that district leaders use the highest quality research, alongside other 
forms of information, to support their decision- making about instruction. But there is 
considerable variability in what and how district leaders use research. This raises the question: 
What conditions enable school district leaders to use research, alongside other forms of 
information, in their decision-making? 


In this report, we focus on organizational routines, which are a central medium through which 
instructional decisions are made in school districts. Routines may matter for research use 
because they bring particular people together at particular moments in the decision-making 
process, shaping what and how decisions are made, and likely the role of research therein. 
However, organizational routines have received little attention in existing scholarship on 
research use. 


We employed a comparative case study approach of four large districts. These districts vary in 
two dimensions - the extent to which they drew on external sources of research, and the 
presence of organizational routines - that prior theory suggested mattered for research use. 
Data collection involved interviews of district leaders on their decision-making in mathematics 
and English Language Arts (ELA) at the elementary level. For this report, we focused on 140 
interviews with district leaders that related to district organizational routines around ELA 
professional development. Our research questions are: 


e How, if at all, do organizational routines structure decision-making? 


e What research and other forms of information do district leaders use in their 
decision-making? 


e How does the structure of organizational routines shape the role of research and 
other forms of information in decision-making? 


Key findings from the study include: 


o Inall four districts, district leaders accomplished the complex work of instructional decision- 
making around ELA professional development by using multiple, interrelated, routines that 
both divided up decision-making into different tasks and also connected decision-making 
between individual routines. 
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o Districts divided-up the disparate work of decision-making into three different types of 
routines: design, deployment, and diagnosis. 


=" Design routines involved district leaders developing a curriculum or instructional 
initiative related to ELA professional development. 


= Deployment routines involved district leaders, and sometimes external partners and 
school staff, providing previously designed professional development. 


=" Diagnosis routines involved evaluating the provision of professional development 
teachers and other staff. 


decision- g. The the number of distin: information sources irangel, 
the ralatlve distribution of information types (palancey. and the degree to which a fewer or 
larger number of district leaders invoked research and other forms of information (spread). 


o The type of the routine, whether it be design, deployment, or diagnosis, influenced what 
information district leaders used. 


o The presence of connections or lack of connections between routines shaped the degree to 
which research use in one routine influenced the work in other routines within a district. 


o Weuncovered a new form of research use, which we call latent use. Latent use occurred as 
district leaders in one routine embedded research in artifacts, which then guided the work 
of leaders in other routines in substantive ways. 
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Introduction 


School districts play a central role in instructional improvement in our nation’s public school.* 
They are not only responsible for implementing state and federal initiatives,* in many states 
they have responsibility for supporting underperforming schools, designing and implementing 
initiatives to improve teaching and learning, providing professional development, choosing 
curriculum materials (or delegating that responsibility to school sites) and, more generally, 
providing management, oversight, and leadership system-wide.? In spite of the proliferation of 
charter schools, conventional public school districts still educate in the neighborhood of 94 
percent of all public school children.* Thus, the decisions that district leaders make have major 
consequences for educational opportunity and outcomes for the majority of students in our 
nation’s public schools. 


Given the consequential nature of these decisions, it is important that district leaders use the 
highest quality evidence and expertise to support their decision-making about instruction. We 
know from the NCRPP national survey of educational leaders that district leaders perceive 
educational research as valuable and useful, and indeed report that they consult research as 
part of their decision-making at high rates. But there is considerable variability in what and 
how they use this research.° We also know that leaders in school districts and a range of other 
public agencies rarely rely solely on research in making decisions. When research is drawn 
upon, it is nearly always in conjunction with other forms of information, including data, 
information about feasibility, prior experience, and expert guidance to name a few.’ This raises 
the question: What conditions enable school district leaders to use research, alongside other 
forms of information, in their decision-making? 


Existing scholarship suggests that variability in research use between and within school districts 
may be a function of individual factors,® organizational arrangements,’ and school districts’ 
connections to outside sources of research, including universities, intermediaries, and 
consultants.*° In this report, we focus on a key aspect of organizational arrangements - 
organizational routines - to understand how research and other forms of information are used 
in school district instructional decision-making. Organizational routines are “repetitive, 
recognizable pattern[s] of interdependent actions, involving multiple actors.’’* District leaders 
engage in designed and emergent routines in the course of their everyday work, from routines 
for designing professional development to structured processes for building the budget, to 
procedures for adopting curricula among others.** Some districts have organizational routines 
in place for decision-making (meeting structures, procedures for selecting materials, protocols 
to guide deliberation, roles devoted to knowledge acquisition, etc.) that serve to bring research 
more routinely into deliberations and shape interpretation and debate.'* Routines such as 
these are important because they bring particular people together at particular moments in the 
decision-making process, shaping what and how decisions are made.“ The structure of routines 
also likely influence what information is available at what stage in the process, bringing in and 
focusing attention on some forms of information in the decision-making process and not 
others. To date, most of the research on organizational routines has been done outside of 
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education. Research at the intersection of routines and public schools has mainly focused on 
the school level with a few recent studies focusing on the school district.° Given that routines 
are the central medium through which decisions are made in school districts, it is surprising 
that they have received limited attention in existing scholarship on research use. 


Using the concept of organizational routines, we analyzed data from our study of decision- 
making in four of our nation’s largest school districts. With funding from the Institute of 
Education Sciences (IES) under the auspices of the National Center for Research in Policy and 
Practice (NCRPP), we undertook an 18-month study of the role of research and other forms of 
information in decision-making in English Language Arts (ELA) and mathematics. In this report, 
we focus on the role of organizational routines in what information district leaders used in their 
decision-making about ELA. We focused on ELA because district leaders in all four districts were 
engaged in decision-making around delivering professional development in elementary ELA 
during the study. 


We find that each district accomplished the work of professional development in ELA through a 
series of interlocking routines. These ELA routines, while including similar types of activities 
across districts, differed in the ways that they related to one another. The nature of the activity 
- whether it was designing, deploying, or diagnosing ELA PD - influenced that nature of 
information that district leaders drew upon. Further, the nature of the connections or lack of 
connections between routines shaped the degree to which research use in one routine 
influenced the work in other routines within a district. Districts varied in the range, balance, 
and spread of information used in decision-making related to professional development in 
English Language Arts. Finally, we uncovered a new form of research use, which we call latent 
use. In the districts in our study, latent use occurred as district leaders in one routine 
embedded research in artifacts, which then guided the work of leaders in other routines in 
substantive ways. 
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Literature Review 


Contrary to the widespread assumption that district leaders rarely use research in their 
decision-making, scholarship on research use has provided evidence that, in fact, district 
leaders do use research and do so with some frequency. For example, our colleagues at the 
National Center for Research in Policy and Practice (NCRPP) surveyed a nationally 
representative sample of school and district leaders and found that 70 to 88 percent reported 
using research “frequently” or “all the time,” depending upon the decision-making activity.*° 
Interview-based studies provide more nuance to this report, showing that district leaders 
conceptualize “research” broadly, including books, peer reviewed journal articles, research- 
informed magazine articles, and even data.’”? When asked to name a piece of research that has 
been useful in their decision-making, district leaders overwhelmingly point to books, most of 
which are targeted to a practitioner audience.*® They also point to state reports, articles in their 
professional association periodicals, research-based tools (such as assessments or textbooks), 
and policy reports prepared by researchers. While they do name peer-reviewed journal articles, 
they are far less salient than other forms of research they draw upon and find useful.?? 


Research is only one of many forms of information that district leaders draw upon as they make 
decisions. Studies have documented the wide range of information that district leaders draw 
upon in the course of their decision-making.2° With the rise of data-informed decision-making, 
achievement, administrative, and implementation data has become particularly salient in 
district decision-making.? District leaders also draw on expert guidance, their own prior 
experience, information from the state, county, or neighboring districts, and anecdote in their 
decision-making.”? While this scholarship is helpful in establishing that indeed research is used 
and the kinds of research that district leaders find useful, we know little about when and how 
research and other forms of information enter into decision-making processes in school 
districts, and how that is shaped by the structure of their decision processes. 


Decision-making in school districts unfolds in complex organizational arrangements. Most 
districts, especially large ones, have highly segmented and departmentalized organizational 
structures. Decision-making related to instruction is often stretched across multiple units in the 
central office and levels of the system. Different district subunits have individuals with different 
disciplinary backgrounds and connections to external sources of research,”? resulting in 
attitudes toward research use that vary systematically by division and level.”4 Patterns of 
within-level (e.g., district office) and cross-level (e.g., between districts and individual schools) 
interaction may support or impede research use in decision-making. 


We know from research in school districts and other public agencies that decision-making is not 
an event but an interactive process involving many people in and across a series of meetings 
(task forces, committees, teams, etc.) and informal conversations” that stretch across system 
units and levels?® and time. It involves habituated patterns of action, meaning, and relations?’ 
that may differ within and across district subunits. Decisions “accrete,” to use Carol Weiss’s 
term, over time in a gradual, nonlinear process.”° 
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Most organizations have routines that structure this decision-making. That is, they have 
repetitive formal and informal ways of working that influence who is involved, when, and how. 
For example, many districts have established processes for adopting new instructional materials 
that involve issuing Request for Proposals, followed by a structured review process, an internal 
decision, and then school board action. Other districts have routines for establishing curricular 
priorities or schools in need of additional support in response to data and other information.”? 
Additionally, organizational research outside of education suggests that organizations often use 
routines to divide-up and coordinate their decision-making work.2° While we know that 
organizational routines play a key role in how decision-making unfolds in an organization, we 
know little about how the structure and organization of these routines influences how research 
and other forms of information enter into and becomes part of instructional decision-making. 


To address this limitation, we asked the following research questions: 


1. How, if at all, do organizational routines structure decision-making? 

2. What research and other forms of information do district leaders use in their decision- 
making? 

3. How does the structure of organizational routines shape the role of research and other 
forms of information in decision-making? 
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Conceptual Framework 


To investigate these questions, we focus on instructional decision-making in elementary English 
Language Arts. All districts engage in instructional decision-making practices as they work to 
support and guide English Language Arts instruction. Carol Weiss reminds us that decision- 
making involves a number of actions: decision makers “recommend, advise, confer, draw up 
budgets, testify, develop plans, write guidelines, report, supervise, propose legislation, assist, 
meet, argue, train, consult”.2? We define instructional decision-making to include when district 
leaders: 1) select policies and programs from external sources, then planning the roll out of 
these programs district wide through piloting and professional development; 2) build their own 
policies and programs by designing them locally; and 3) maintain instructional policies and 
programs, adjusting and redesigning them over time. We focus on when and how research and 
data enter decision-making processes when district leaders are selecting, designing, and 
maintaining English Language Arts policies and programs. 


We define research as the product of an activity in which 
people employ systematic, empirical methods to answer a 
specific question.** Research can enter into a deliberation 
in many ways: 1) when individuals invoke findings from a 
specific research study, book or other publication; 2) when 
they cite a researcher or research organization; 3) when it is 
built into research-based tools such as curricula or 
assessments; or 4) when they make broad “research says...” 
claims.?? We define data as measures or specific outcomes 
related to student achievement, school performance, 
teaching practice, and program implementation, 

among others.*4 


Research: The product of an activity 
in which people employ systematic, 
empirical methods to answer a 
specific question. 


Data: Measures or specific 
outcomes related to student 
achievement, school performance, 
teaching practice, and program 
implementation. 


When and how research enters into the practice of decision-making is likely influenced by a 
district’s organizational routines. Consistent with our earlier definition, to count as an 
organizational routine, interaction patterns must be repeated over time, 

recognizable to organization members, and involve two or more organizational members. 
Routines can either be emergent - developing in and from everyday practice in organizations - 
or intentionally designed by decision makers.*° Organizational routines are both the medium or 
vehicle for decision-making practices (i.e., decisions about mathematics and ELA are made in 
and through organizational routines) and an outcome of decision-making (i.e., routines may be 
designed, altered, eliminated, and evolve during the decision-making process). 


Routines have ostensive and performative aspects.*° The ostensive aspect refers to the 
idealized form of the routine — the broad script for how the routine works as described in 
official documents or as told to us in interviews by school district staff. For example, district 
office personnel have a script for instructional walkthroughs with school leaders. The 
performative aspect refers to how the routine actually happens in practice in particular places 
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at particular times. For example, it may be that in enacting the walkthrough routine, 
participants deviate from the structured protocol or that they blur the line between different 
roles. In this report, we focus on the ostensive aspect of the routine as a first step in our 
investigation of the role of routines in the use of research and other forms of information. 
Attending to the ostensive is critical, as the ostensive version of the routine structures and 
guides how district leaders actually do their work. For instance, the ostensive walkthrough 
script may focus what district and school leaders look for in their walkthroughs on inquiry- 


based instruction. 


Routines: Repetitive, recognizable 
patterns of interdependent actions, 
involving multiple actors. 


Decision-making routines: Bring 
people together at particular 
moments in the decision-making 
process, shaping what and how 
decisions are made. 


Routine clusters: Two or more 
interdependent routines that, 
together, support the completion 
of a shared task. 


Decision-making in complex organizations rarely 
happens in the context of a single routine. For 
instance, in the previous example of adopting 
instructional material, there may be separate but 
interlocking routines for creating the Request for 
Proposals for vendors, for analyzing different 
curricula, for piloting them, for gathering data from 
a pilot and making recommendations, and for 
putting the adoption before the School Board for a 
vote. Scholars who study organizational routines 
define a collection of interrelated routines as a 
“routine cluster.” Routine clusters involve two or 
more interdependent routines that, together, 
support the completion of a shared task.*” Routines 
are interdependent when one routine sets the 
context for action in another routine such that the 
actions in one routine are consequential for the 
actions in another routine.*® 


Routine clusters enable organizations to manage their complex work by dividing up that work 
across people and subunits. They facilitate a division of labor by allocating different parts of the 
decision-making process to different people.*? At the same time, routine clusters, and in 
particular, linkages or connections between interdependent routines, allow for the 
coordination of the different but interdependent work undertaken in different routines. One 
main way that interdependent routines are linked is when a routine produces an output that 
triggers the start of the successive routine. An output could include decisions and material 
artifacts that were produced by one routine and are handed off to a subsequent routine.*° For 
instance, returning to the adoption discussion. The outcome of a routine that starts the 
curriculum adoption process might be a Request for Proposals. This output is handed off to and 
initiates a second routine where district leaders use this Request for Proposals to gather 
competitive bids from various textbook publishers. In this way, actions in one routine inform 
actions in the successive routine. They also likely influence what information is in play and how 
that information influences the nature of the work at different points in the different sub- 


routines in a given decision cluster. 
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We hypothesize that the following dimensions of a routine cluster likely influences what and 
how research and data is invoked and used in district decision-making: 1) who is involved in the 
routines, as routine participants can bring research and other types of information into the 
routine; and 2) the presence of connections between interdependent routines, as these 
connections may enable tools, resources, and information to flow between different routines.** 


Technical Report No. 5 « | 12 
National Center for Research in Policy and Practice 


Study Design/Methods 


To investigate these issues, we draw on data from an 18-month study of instructional decision- 
making in four of our nation’s largest school districts. To sample districts, our NCRPP colleagues 
administered a survey to district leaders in 32 of the nation’s largest school districts in states 
that had adopted the Common Core State Standards in mathematics and English Language 
Arts.*2 We sought districts that varied on two key dimensions that prior empirical and 
theoretical work suggested would be associated with research use. First, we sought districts 
that varied in the extent to which they drew on external sources of research to inform their 
instructional decision-making.*? We sought districts where external partnerships with 
universities and other research entities were plentiful and figured prominently in district 
decision-making, and districts where they were not. Second, we sought districts that varied in 
terms of the prevalence of organizational routines that enabled research use in decision- 
making.” We reasoned that districts with organizational routines that explicitly cued research 
or data use might be more likely to incorporate these forms of information in their decision- 
making than those that did not. We sought school districts where staff reported that there 
were a high number of routines that support research use and districts where staff reported a 
low number of routines that support research use. Analysis of the survey data showed that the 
two dimensions were highly correlated (.63) prompting us to focus on their interaction in 
purposefully sampling four districts. Our final sample included two districts that were high in 
connections to external resources and the prevalence of organizational routines to support 
research use (Hawthorne and Magnolia) and two districts that were low in both dimensions 
(Willow and Juniper).*° Within each district, we also sampled four schools to investigate the 
relationship between district decision-making and school decision-making. We do not use that 
data in this report. 


The four school districts were located in different regions of the U.S., including the South, the 
Midwest, and the Western U.S. Three of the districts were large urban school districts, and one 
was a large county district that included suburbs and rural areas close to a major metropolitan 
area. While senior leadership in one district has been stable, there was more turnover in senior 
leadership (i.e., superintendent, cabinet-level) in the other three districts. As displayed in Table 
1, the demographics of the student population served by the districts varied by race, ethnicity, 
home language, and income. 
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F : : Native : 
a ene Hispanic Black/ pinelican asian Qualify for Language 
or Latino . . Indian and . Free and other than 
students White African Asian | and other . 
(of any . Alaska ae Reduced English spoken 
(rounded) American ; Pacific ea 
race) Native Lunch at home 
Islander 
Hawthorne 100,000 10% 50% 35% 1% 5% 02% 62% 12% 
Willow 95,000 26% 16% 47% 5% 5% 01% 68% 20% 
Magnolia 180,000 29% 29% 31% 1% 9% 03% 56% 34% 
Juniper 90,000 56% 28% 11% 6% 6% .6% 71% 41% 


*Source of data: National Center for Education Statistics 2013-17 dataset 
** Source of data: individual district websites 
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Table 1: District demographics* 
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Data Collection 


We collected data over 18 months in each district.*° We focused our efforts on decision-making 
in mathematics and English Language Arts (ELA) at the elementary level,”’ as they are the most 
central subjects in elementary schools. We then interviewed district leaders who were involved 
in decision-making in ELA and mathematics. Since decision-making in subject areas is typically 
stretched across departments in a district, we identified district leaders involved in such 
decision-making in departments related to content areas, leadership (those supervising 
schools), special education, English Language Learners, and research and accountability. District 
leaders were interviewed 2-4 times across the 18 months. Table 2 shows the number of people 
and number of interviews for each district. We also collected artifacts related to decision- 
making, including handouts, policy documents, googledocs, plans for professional 
development, and copies of the research and data leaders used. 


Table 2: Number of interviews by district 


Number of | Number of 

IVs Participants 
Hawthorne District 113 43 
Willow District 70 36 
Magnolia District 104 45 
Juniper District 107 53 


Data Analysis 


To investigate our research questions, we began by identifying all interview data related to ELA 
decision-making. In each district, leaders worked on a variety of policy, planning, and program 
implementation related to ELA. We chose one domain of ELA work to focus on for this analysis - 
district professional development - because it is a central decision-making activity for district 
leaders*® and because all four districts were engaged in decision-making in this domain during 
the time of our study thereby facilitating comparison across districts. Table 3 shows the number 
of interviews we analyzed related to ELA professional development across the four districts. We 
coded all interview transcripts for information about organizational routines related to ELA 
professional development. 
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Table 3: Interviews about ELA professional development decision-making 


Number of Interviews in the Routines Analysis 


Hawthorne District 23 
Willow District 27 
Magnolia District 26 
Juniper District 64 


To address research questions 1 - How, if at all, do organizational routines structure decision- 
making? - we wrote district-specific memos that described all routines related to professional 
development in English Language Arts. We focused on capturing the ostensive aspect of 
routines; that is, the generalized or abstract idea or script of the routine.*? Although we 
primarily drew on interview data, we also triangulated interviews with artifacts related to 
professional development (e.g., power points, agendas, workbooks). Interviews and artifacts 
have been used in previous work to study the ostensive aspect of the routine.°° First, we 
identified individual routines in each district. To count as a routine, it had to be a recurring 
activity where district leaders could complete the work without checking on what was 
happening in other routines! and where the work was organized around a single sub-task (e.g., 
designing professional development). If the work focused on different sub-tasks, for example 
designing professional development versus delivering it, we considered it to be separate 
routines.>* 


We then analyzed all routines, focusing on three dimensions: 1) the type of routine; 2) who was 
involved and what they did; and 3) if and how individual routines in a given district interrelated 
with each other. Regarding the type of routine, we worked inductively to develop a set of 
codes that identified the different kind of work done in different routines. We identified three 
type of routines: design, diagnosis, and deployment. Design routines involved district leaders 
developing a curriculum or instructional initiative related to ELA professional development. 
Deployment routines involved district leaders and sometimes external partners and school staff 
providing this professional development. Diagnosis routines involved evaluating the provision 
of professional development to teachers and other staff. 


To analyze who was involved in what ways, we identified and tracked key participants in 
different routines, including their role and department. To investigate the ways in which 
individual routines within a district were related to one another, we focused on the degree to 
which individual routines interrelated within clusters. We drew on prior research°? that 
suggested that routines within clusters might be interrelated through input-output 
relationships, where inputs triggered the start of the routine and outputs signaled the end of 
the routine. For example, a routine where district leaders designed a professional development 
session might produce an output - a slide deck - that would serve as the input of the 
subsequent routine, where district leaders used this slide deck to facilitate professional 
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development to teachers. We also identified multiple routine clusters in some districts. Routine 
clusters are separate from one another when they enable the accomplishment of different lines 
of work (e.g., literacy professional development, curriculum adoption) that can operate 
relatively independently from each other.** For example, while the adoption of a new literacy 
curriculum will likely influence work on literacy professional development, these two lines of 
work can be accomplished mostly independently of one another. In contrast, routines within a 
cluster are more tightly interdependent in that they rely on multiple input-output connections 
between routines to coordinate work. 


Accordingly, we analyzed the relationships between routines, establishing whether there was 
an input-output relationship and what the inputs and outputs were. We also developed visual 
maps of the structures of routine clusters in each district, including the individuals and actions 
that comprised each routine, and how routines interrelated with other routines. Finally, we 
crafted memos for each district that provided rich descriptions of the routines, including the 
components, individuals, activities, and artifacts that made up each routine.°° 


To address research question 2 - What research and other forms of information do district 
leaders use in their decision-making? - we developed a coding system to identify and categorize 
instances where leaders discussed the information they used in decision-making routines about 
ELA PD. We used a mix of deductive and inductive methods to develop codes. Based in 
literature on research and evidence use, we began with a focus on documenting the research 
and data that leaders referred to. Other codes were generated inductively, as we read 
interview transcripts and examined how leaders talked about sources of information and 
guidance. Leaders used and discussed a wide range of publications, types of data, as well as 
individuals from outside organizations who provided resources in the form of ideas, texts, and 
access to knowledge resources outside of the district. Our coding scheme is represented in 
Table 4. 


To investigate what kinds of information leaders drew on, we coded references to distinct 
sources of information, rather than for frequency of or use. Our focus was on identifying the 
number of unique pieces of information that district leaders drew upon to inform their 
decision-making. For that reason, if multiple district leaders identified the same source of 
information (for example, a specific book), we counted that as a single source of information. At 
times, district leaders referred to “data” or “research” generally. Because we did not have 
enough information to tell if district leaders were referring to similar or different data or 
research, we counted each separate general mention of “data” or “research.” After coding 
interviews for references to information used in ELA PD decision-making, we displayed patterns 
of information use and wrote analytic memos to identify patterns within and across districts. 
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Table 4: Research and Other Forms of Information Codes 

Code Definition 

Student assessment data (district and state) 
Other diagnostic and formative student assessments 
Teacher surveys (post-PD and needs assessments) 
General “data” 

Professional trade book 

Curricular or instructional resource or tool 
Academic publication 

General “research” 

University 

Professional Association 

Other 

Academic Researchers 

Consultants 


Data 


Research (publications) 


Organizations 


Individuals 


To answer research question 3 - How does the structure of organizational routines shape the 
role of research and other forms of information in decision-making? - we brought together our 
analysis of routines, and our analysis of information named in these routines. We wrote cross- 
district memos to examine similarities and differences in the relationship between the structure 
of routines and information use across districts. Our cross-district memos focused on analyzing 
the relationship between our two dimensions of interest - the type of routine, and how 
routines within a cluster interrelated - and our outcome of interest - patterns of information 
use. 
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Findings 


In this section, we compare and contrast the four districts with respect to their use of 
organizational routines for decision-making around English Language Arts professional 
development, and the role of research and other forms of information therein. We find that 
district leaders in all four districts accomplished the complex work of instructional decision- 
making through the use of multiple interrelated routines that were often organized into routine 
clusters. 


Turning to information use in these routines, district leaders in all four districts described 
drawing on four types of information in these decision-making routines and routine clusters: 
research, data, people, and organizations. Information use varied across districts in the percent 
of district leaders that explicitly named any information, the range of distinct sources that 
district leaders named, and the type of information was most frequently named (e.g., research 
versus data versus people). 


Finally, two dimensions of routines and routine clusters - the type of organizational routine and 
how routines were connected to each other - played a key role in shaping how district leaders 
used information in ELA PD decision-making, contributing to both variation within and across 
districts. We also identified a new form of research use, which we termed “latent use.” 
Heretofore unacknowledged in discussions of different kinds of research use,*° latent use 
occurred when district leaders in one routine embedded research into artifacts which then 
guided the work of district leaders in other routines. 


The Work of ELA Professional Development 


District instructional decision-making is not an event but rather a process that unfolds over 
weeks, months and even years, often involving multiple interdependent decisions. For the 
districts in our study, designing and rolling out professional development in ELA involved an 
array of decisions, including deciding what aspects of ELA instruction to focus on, what content 
to cover, whether and how to differentiate the materials by grade levels, how to deliver 
professional development opportunities to schools, how to address the needs of particular 
student populations such as ELL, the needs of particular schools, and the list goes on. These 
interdependent decisions took considerable time to negotiate, requiring the involvement of 
multiple district leaders with different specializations, often from different district office 
subunits. Sometimes external partners and school staff were also involved in the decision- 
making process. 


We found that district leaders accomplished this complex instructional decision-making process 
by using inter-related routines that divided up the work into different tasks and also created 
connections between individual routines to coordinate the work. Each of the four districts 
constructed routines and routine clusters to make decisions about professional development in 
ELA in different ways. The different structure of these routines, in turn, had implications for 
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what and how they used research and other forms of information in their decision-making. In 
this section, we provide a brief overview of the structure of each district’s routines and routine 
clusters for decision-making about ELA professional development. 


Hawthorne District. In Hawthorne District, district leaders used what they called “Quarterly 
PDs” to train school coaches and teachers on instructional approaches for the newly adopted 
ELA Common Core State Standards. As one district leader explained: “the curriculum maps are 
broken into ... four quarters. Then in each quarter, the curriculum specialists and their teams 
are providing PD for teachers and coaches around the standards on that section of the 
curriculum map ... sharing best instructional practices.” 


District leaders used five interrelated routines to organize this work. In the Literacy Leaders 
Designing Quarterly PD routine (labeled as routine 1 in Figure 1 below), a small team of central 
office ELA leaders used slidedecks and other materials to develop rough outlines for Quarterly 
PD, including content, activities, tools, books, and instructional strategies. This routine 
produced the output of Quarterly PD designs that were embedded in slide decks and 
accompanying resources. This output then served as an input to the subsequent Teacher 
Leaders Designing Quarterly PD routine (labeled as routine 2 in Figure 1), where literacy leaders 
met with teacher leaders, or what Hawthorne District called resource teachers, who then took 
the next pass at fleshing out these PD designs. This routine produced an output of revised PD 
slidedecks and other materials, which served as an input to the Reviewing and Revising 
Quarterly PD Routine (labeled as routine 3 in Figure 1), where resource teachers and ELA 
leaders then reconvened to provide feedback and finalize these PD designs. 


This output of PD slide decks and accompanying materials then served as the input for the 
successive routine, the Facilitating Quarterly PD routine (labeled as routine 4 in Figure 1), where 
the same team of ELA district leaders and resource teachers facilitated these PDs to school 
coaches on a quarterly basis. The Facilitating Quarterly PD routine produced two outputs: 
trained school coaches and completed Quarterly PD. The output of trained school coaches 
informed the provision of literacy PD in schools, where these school coaches would provide 
professional development to teachers and other school staff. 


The output of completed PD served as the input to the successive district routine - the 
Evaluating Quarterly PD routine (labeled as routine 5 in Figure 1), where the same literacy 
leaders evaluated the implementation of professional development. Specifically, literacy 
leaders got feedback from school coaches on completed Quarterly PD via exit slips and needs 
assessment surveys. This feedback data from school coaches then served as the input to the 
next iteration of the Designing Quarterly PD routine, where literacy leaders would use this 
feedback to inform future PD design efforts. Taken together, these three routines formed a 
routine cluster because they represent a grouping of closely interrelated routines that are 
organized around a common task: in this case, the provision of Quarterly PD.°’ 
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Figure 1: Hawthorne District ELA PD Routines 
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Willow District. ELA PD decision-making involved ELA leaders, sometimes in partnership with an 
external partner Learning Supports Partner, designing and providing professional development 
around foundational literacy to school leaders and teachers. While Hawthorne District’s ELA PD 
work involved only one routine cluster, ELA PD in Willow District involved a routine cluster plus 
two separate organizational routines (See Figure 2). PD Institutes focused on training teacher 
leaders (what Willow District called school support teachers or SSTs), principals, and assistant 
principals about foundational literacy (e.g., teaching phonics and vocabulary), with the 
expectation they would train teachers. A Literacy Teacher Leader explained, “it’s kind of that 
train the trainer model where we write the PD, we give it to the SSTs, we walk through the 
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notes and the questions and if an SST has further questions, they invite us out to their school 
before they have to actually deliver it.” 


Decision-making about the PD Institutes involved three interrelated routines. In the Designing 
PD Institutes routine (labeled routine 1a in Figure 2), a cross-departmental group of district 
leaders that called themselves the Reading Foundations Design Team met regularly to design 
the PD Institutes. This cross-department group involved staff from the Curriculum and 
Instruction department, Early Childhood Education department, Leadership department, the 
Grant Supports Office (a specialized office responsible for supporting teaching and learning in 
schools funded by external grants), and school staff, among others. District leaders presented 
their professional development designs to the broader team for feedback, and then made 
revisions to these designs. The Designing PD Institutes routine produced outputs of Institute 
power points and accompanying resources, such as facilitator guides, which then served as an 
input to the Facilitating PD Institutes routine (labeled routine 2a in Figure 2), where the Reading 
Foundations Design Team, in addition to the literacy leaders from the Curriculum and 
Instruction department, used these slide decks and accompany materials to guide their efforts 
to provide PD to school support teachers (SST), principals, and assistant principals. The 
Facilitating PD Institutes routine produced two outputs: trained facilitators and completed PD 
Institutes. The first output - trained facilitators - served as an input to School Routines, where 
SSTs would provide school-level professional learning to school staff. The second output - 
enacted PD - served as an input to the Evaluating PD Institutes routine (labeled routine 3a in 
Figure 2), where literacy leaders evaluated the enacted PD Institutes by administering surveys 
at the end of each PD, and then reviewing these surveys. The output of this routine - feedback 
from surveys - served as an input to the Designing PD Institutes routine, where literacy leaders 
used this feedback to inform their PD design. Taken together, these three routines were 
connected through input-output relationships that all centered on a common task of designing 
and facilitating PD Institutes. 


Willow district leaders also used two other routines for decision-making about foundational 
literacy PD work that were not directly connected to the PD Institutes Cluster but occurred 
concurrently alongside this cluster. In the Designing Online PD Modules routine (labeled routine 
1b in Figure 2), the literacy leaders created online Foundational Literacy PD modules, making 
them available on the district’s employee intranet resources site for teachers to use and discuss 
in weekly grade-level meetings. The output of this routine was online PD modules for teachers 
to use in the school-level routines. Finally, in the Delivering LSP Site-Based PD routine (labeled 
routine 1c in Figure 2), the ELA team surveyed schools to identify those needing literacy support 
and select them for site-based Foundational Literacy PD that was provided by Learning 
Supports Partners. Staff from Learning Supports Partners then worked with select teachers to 
provide literacy support. The output of this routine was trained teachers. These routines were 
not a part of the PD Institutes Cluster because they were not directly connected through input- 
output relationships. 
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Figure 2: Willow District ELA PD Routines 
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Magnolia District. ln Magnolia District, ELA PD decision-making centered on training teachers 
and instructional leaders to implement the Curriculum for Classroom Writing (CCW) produced 
by Education University that district leaders had previously adopted. Magnolia District’s ELA PD 
decision-making process was organized into a routine cluster made up of five routines that 
operated in parallel. This cluster centered on providing professional development to support 
CCW implementation (see Figure 3). Four literacy leaders co-facilitated these routines, 
sometimes involving staff from Education University, the main external partner in the CCW 
work. 


The quarterly Designing Teacher Leadership Training (TLT) routine (labeled routine 1a in Figure 
3) happened four to five times a year and involved central office literacy leaders working 
together with the help of teachers called “thinking partners” and sometimes staff from the Staff 
Development Office, to design professional development sessions for a select group of teacher 
leaders, who would serve as instructors for TLT. This group of teacher leaders were part of the 
district’s Teacher Leadership Team and worked closely with the district’s ELA office to 
communicate district messages regarding literacy instruction to schools and provide support for 
teachers. These TLT PD designs involved the creation of lesson plans and leadership training 
protocols. Literacy leaders also trained TLT teacher leaders around PD design materials. 


The Designing TLT Routine produced two outputs - TLT PD lesson plans and training protocols 
and trained TLT teacher leaders - that then served as inputs to the Facilitating TLT routine 
(labeled 2a in Figure 3). The Facilitating TLT routine involved trained teacher leaders using these 
TLT lesson plans and training protocols to train school staff, with the support of central office 
literacy leaders. The output of this routine was trained teachers and completed TLT PD. Upon 
the completion of the TLT PD, literacy leaders would again initiate the Designing TLT Routines in 
preparation for the next TLT PD. 


At the same time, literacy leaders also engaged in the Designing Insights routine (labeled 
routine 1b on Figure 3) five to six times a year, which involved literacy leaders training teacher 
leaders as a team for what the district called Insights PD, with literacy leaders recruiting a 
different set of teacher leaders than those being trained for TLT. These teacher leaders worked 
together on teams to plan and draft PD design together with support from ELL and SPED 
consultants. During this PD design process, these teams got familiar with, and discussed the 
content of the CCW. Literacy leaders then vetted teacher leaders’ proposed plans and provided 
feedback, which the teachers leaders would then incorporate. 
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Figure 3 : Magnolia District ELA PD Routines 
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The Designing Insights routine produced two outputs - trained teacher leaders and Insights slide 
decks, hand-outs, and other materials - that served as inputs to the Facilitating Insights routine 
(labeled routine 2b on Figure 3). In the Facilitating Insights routine, teacher leaders trained in 
the Designing Insights routine facilitated PD for teachers in their schools, with trainings tailored 
toward each grade level. Teacher leaders also used these Insights slide decks and other 
materials to guide their delivery of PD in the Facilitating Insights routine. Sometimes Education 
University staff or other external partner organizations would provide keynote address during 
Insights. The outputs of this routine were trained teachers and completed Insights PD. Upon 


Technical Report No. 5 « | 25 
National Center for Research in Policy and Practice 


the completion of the Insights PD, literacy leaders would initiate the Designing Insights routine 
again in preparation for the next Insights PD. 


Finally, the Education University Institutes routine (labeled routine 1c on Figure 3) involved 
Education University project staff conducting one-day trainings around CCW kits a couple of 
times a year. Sometimes Education University also came to Magnolia District to facilitate a 
week of mini institutes for different grade levels. Education University staff would coordinate 
logistics with district staff. Teachers, teacher leaders, and school-level administrators then 
signed up for and attended these Institutes. The outputs of this routine were trained teachers, 
teacher leaders, and school-level administrators, in addition to completed Educational 
University Institutes PD. 


Magnolia district leaders used these five routines, operating in parallel, to disseminate core 
messages about effective writing instruction to teachers. They also trained a network of 
teacher leaders and administrators to support teachers with using the CCW to teach writing. 
Here, the trained teachers, teacher leaders, and administrators all served as inputs to support 
the provision of school-level PD around the CCW, where trained teacher leaders and 
administrators provided CCW professional development in schools. Then teachers taught the 
CCW in their classrooms. Literacy leaders and some assistant superintendents observed the 
implementation of the CCW in classrooms. They brought back these insights to discuss with the 
literacy leaders and the leadership department. This output - insights from observations of 
classroom instruction - served as the inputs to the Designing TLT PD and Designing Insights PD 
routines, informing literacy leaders’ efforts to design future TLT PD and Insights PD. 


Juniper District: Juniper District’s Foundational Literacy PD decision-making was accomplished 
through a routine cluster made up of six interrelated organizational routines (see Figure 4). 
Juniper District’s literacy leaders designed over 20 hours of PD content divided into eight 
different modules to train Early Literacy Specialists. The expectation was that Early Literacy 
Specialists would, in turn, train school staff. As one literacy leader explained, “Everything is 
turnkey. Everything is ready to go, differentiated materials by grade level, by language, scripted 
out, like, ‘Here’s what you say.’...Slide decks. The materials are extensive.” 


In the Designing Foundational Literacy PD routine (labeled routine 1 in Figure 4), a small team 
of literacy leaders and their external partners School Improvement Partner and Career 
Preparation Partner discussed what Foundational Literacy should look like. The external 
partners then developed a first draft of this slide deck. This routine produced an output 

of a slide deck, which then served as an input for the Literacy Leader Reviewing Foundational 
Literacy PD routine (labeled routine 2 in Figure 4), where literacy leaders discussed the contents 
of this PD slide deck and provided feedback to the external partners School Improvement 
Partner and Career Preparation Partner, who made another round of revisions to the slide deck. 
This routine produced an output of another slide deck, which then served as an input to the 
Cross-Department Content Reviewing Foundational Literacy PD routine (labeled routine 3 in 
Figure 4). In this routine, district leaders from across different departments collectively 
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reviewed the slide deck and provided feedback to external partners through a phone call. These 
departments included the literacy Curriculum and Instruction department, the Assessment, 
Research, and Evaluation department, Student Services, and Early Childhood Education, among 
others. The output of this routine - feedback from multiple cross-department district leaders, 
and the PD slide deck - served as an input to the Revising Foundational Literacy PD routine 
(labeled routine 4 in Figure 4), where external partners School Improvement Partner and Career 
Preparation Partner used this feedback to make revisions to the slide deck. Then literacy 
leaders made revisions to, and finalized the slide deck, which served as the output of this 
routine. 


Figure 4: Juniper District ELA PD routines 
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The finalized draft of the module slide deck served as the input to the Training District 
Facilitators routine (labeled routine 5 in Figure 4), where literacy leaders used the module slide 
decks to inform their efforts to provide professional development to other district leaders. In 
the Training Early Literacy Specialists routine (labeled routine 6 in Figure 4), the district 
facilitators were then responsible for facilitating these PDs to early literacy specialists, but also 
school leaders, teachers, and paraprofessionals. The routine produced two outputs: completed 
PD and trained early literacy specialists. The trained early literacy specialists then conducted 
literacy PD in schools. Both completed PDs - those facilitated by district facilitators and those 
facilitated by early literacy specialists - then served as the input to the Evaluating Foundational 
Literacy PD Routine. 


Finally, in the Evaluating Foundational Literacy PD routine (labeled routine 7 in Figure 4), 
literacy leaders sent out surveys to gather feedback from teachers and other school staff about 
these PDs. The output of this routine - survey feedback on completed PD - then served as the 
input to the Design of the Module Slide deck, where literacy leaders and external partners may 
have drawn on these changes to make changes to the design of the PD module slide decks. 


To summarize, in all four districts, interrelated organizational routines structured instructional 
decision-making in two ways. First, routines enabled district leaders to divide up the complex 
work of ELA professional development decision-making into different tasks. Second, input- 
output relations helped to coordinate decision-making work undertaken in different routines. 


Sources of Information in ELA Professional Development Routines 


Consistent with prior research, district leaders across the four districts described drawing on 
multiple sources for information in their decision-making in the course of their ELA PD routines 
and routine clusters. They referenced research publications, but also identified data, 
individuals, and organizations as sources of information that informed their work. 


Of these sources of information across districts, leaders referred to data more than any other 
source, accounting for over a third of the information sources that district leaders discussed as 
informing their decision-making work (See Table 5). District leaders were most likely to refer to 
state and district student assessment data. For example, one ELA content leader explained how 
she worked with assistant superintendents in the district to review student assessment data to 
decide the content focus of upcoming professional development sessions: 


We’ve just done a data analysis of our assessments districtwide. We are going to meet 
with the Assistant Sups... share the data, and that is what determines our next PDs that 
we do. [So for instance...] district-wide, the standard that had the lowest percent of 
questions answered correctly was [the Common Core standard on using language that 
pertains to time, sequence, and cause/effect in informational texts]. That was a need for 
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third grade. So, we’re going to provide some sort of PD electronically across the district 
to meet the need, based on how [students] did on that last assessment. 


In addition to student assessment data, district leaders also described relying on other forms of 
data including needs assessment surveys and PD evaluation surveys collected from teachers. A 
number of district leaders also made general statements about relying on “data,” without 
naming anything specific. 


Individuals were the second most prominent source of information named by district leaders, 
accounting for 27.9% of information sources that they referenced with respect to their ELA 
decision-making. The vast majority of these individuals (13 out of 17) were academics. That is, 
they had a PhD and/or were associated with a research organization. For example, Nell Duke 
(University of Michigan), Lucy Calkins (Teachers College), Isabel Beck (University of Pittsburgh), 
and Carol Tomlinson (University of Virginia) were among those individual academics named by 
district leaders. The remaining individuals named as information sources were consultants of 
various types. Consultants tended to either be associated with the vendors from whom district 
leaders had purchased curricular or instructional materials or were authors of practitioner 
publications. Three of the 17 individuals named by district leaders were partnering directly with 
districts. 


Table 5: Unique sources of information informing ELA Professional Development Across 


Districts 
N % 

Data 22 36.7% 
Student assessment data (district and state) 9 
Feedback surveys 3 
General “data” references 10 

Individuals 16 26.7% 
Academic researchers 12 
Consultants 4 

Research 14 23.3% 
Professional trade books 7 
Curricular/Instructional Resource or tool 1 
Academic publication 1 
General “research” 5 

Organizations 8 13.3% 
University 3 
Professional Association 1 
Nonprofit 1 
Vendor 3 

TOTAL 60 
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Research, in the form of publications or general references to research, made up approximately 
23% of the sources of information named by leaders. Some of these publications were written 
by academic researchers, but most were not. Consistent with research from our NCRPP 
colleagues,°® professional trade books made up the largest number of publications that district 
leaders drew upon (7 out of 14). 


One district - Magnolia District - drew heavily on their research-based instructional materials in 
their decision-making, in this case the Curriculum for Classroom Writing developed by 
Education University. As one district leader explained: 


We’ve been a Balanced Literacy district, but there were parts of it that we needed 
tightening up... It’s mapped out with a writer’s workshop and our mini lessons were 
pretty solid, but a place that our teachers were saying they needed some support with 
was in the art of conferring with writers. The [Curriculum for Classroom Writing] provide 
some guidance on what that might look like, and how you go deeper to really change 
what a student is thinking, and to push them along to really develop and fine-tune their 
writing. 


Notably, only one district leader - in Juniper District - named a peer-reviewed publication: The 
National Research Panel Report Teaching Children to Read (2000), which is a meta-analysis of 
studies related to early literacy. 


Finally, four district leaders made references to drawing on research generally, rather than 
referring to a specific piece of research. For example, an ELA leader in Hawthorne District 
reflected that she wanted to focus on some foundational literacy practices in the classroom, 
which would “take [PD] in a different direction and be more research-based instead of just 
addressing what’s going to be on the proficiencies.” 


Thirteen percent of information sources were organizations (8 out of 61). Three were 
universities (University of Kansas, Teachers College Columbia University, and Florida Center for 
Reading Research at Florida State University), one was a professional association (ASCD), three 
were vendors (Houghton Mifflin, PARCC, and SchoolKit), and one was a nonprofit (Uncommon 
School) that provided educational products or services to the district. 


Information Use in Decision-making Varied Between School Districts 


Information use varied by school district in three ways. First, districts relied on different 
numbers of unique sources of information, which we call range. Second, they varied in the 
balance of those sources, by which we mean the distribution of these information sources 
across different types of information (e.g., research versus data versus people). Third, districts 
varied by the percent of district leaders that explicitly named any information source in the 
course of their work, which we call spread. 
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First, districts varied in the range of information they used in their decision-making about ELA 
professional development. District leaders in Hawthorne District and Juniper District described 
relying on a large range of information sources - 20 and 25 respectively - whereas leaders in 
Willow District and Magnolia District identified a smaller range of information sources - six and 
10 respectively. While a large range of data sources may introduce multiple and diverse 
information into the decision-making mix increasing access to diverse ideas, a narrow range 
may contribute to more focused and coherent conversation. 


Second, districts also varied in the balance of information sources: the distribution of 
information sources across information types. In Hawthorne District, data, individuals, and 
research accounted for 90% of all data sources (18 out of 20) mentioned, with organizations 
receiving much less attention (see Table 6). District leaders relied on individuals most heavily, 
accounting for 35% of all information sources (7 out of 20). Most of the individuals mentioned 
were authors of popular trade books related to literacy instruction. For example, one district 


leader explained: 


Debbie Diller. We always read her. We’re always reading Marzano. We’re always 
following—I just lost her name, and we love her. Calkins...Jan Richardson, on her guided 
reading. We’re always looking for the books that we see teachers running to, and then 
we want to pull it and say, “Okay, what’s going on?” 


Hawthorne District also drew heavily on data, which comprised 30% of all information sources. 
They referenced a range of data, including a student writing assessment, information from 
teacher surveys, and four general mentions of data. District leaders referenced five different 
sources of research. All were trade books related to literacy. Finally, they referenced two 
organizations: a university and a professional organization. 


Table 6: Number and percent of sources of information by information type 


Hawthorne Willow District Magnolia Juniper District 
District District 

N % N % N % N % 
Data 6 30% 2 33% 3 30% 11 44% 
Individuals 7 35% 0 0 5 50% 5 20% 
Research 5 25% 2 33% 1 10% 6 24% 
Organizations 2 10% 2 33% 1 10% 3 12% 
Total 20 6 10 25 


While individuals were the most prominent source of information in Hawthorne District, they 
were not mentioned at all in Willow District. Instead, data, organizations, and research were 
equally weighted as information sources for district leaders in Willow (See Table 6). Two out of 
six information sources were data. One was a reading assessment administered to early grades 
children and the other was survey data that schools were collecting about children’s reading 
preferences. Beyond data, district leaders in Willow District also drew on a commercial 
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publisher. A leader who designed PD described drawing on basal readers by Houghton-Mifflin: 
“we have our Houghton-Mifflin - we have the basal reader, and we have had it for eight or nine 
years... We went through the Houghton-Mifflin teacher’s edition and pulled out the 
foundational reading pieces that were non-negotiable. You [teachers] must do this every day.” 
Finally, a single district leader made an allusion to being expected to regularly use research 
evidence: “We always want to use research. An example of that would be for our interventions, 
[they have to be] scientifically research-based interventions.” 


For Magnolia District, the balance of information sources was tipped strongly toward 
individuals with data receiving far fewer mentions, and even fewer still references to 
organizations and research. Magnolia District relied even more heavily on individuals than 
Hawthorne District with 50% or 5 out of 10 information sources being individuals. Like 
Hawthorne District, some of the individuals they relied on for information were academics who 
were authors of popular trade books related to literacy. Others were academics who they 
worked with directly as part of their partnership with the Education University around the 
implementation of the Curriculum for Classroom Writing. Further, the organization they drew 
upon was Educational University, the institutional home of that project and the sole publication 
that they drew upon was the curricular materials developed by the project. Finally, district 
leaders drew on a few distinct kinds of data, with data accounting for 3 out of 10 sources of 
information. One data source was “weighted school assessment” and the others were general 
mentions of data. 


Finally, Juniper District drew on data more than other sources of information. Eleven out of 25 
of their information sources were either discrete sources of data (7) or general mentions of 
data (4). Most of these data sources provided information about student performance on state 
accountability tests or district multiple choice interim assessments. Juniper District leaders also 
drew on teacher survey data to inform their work. As one district leader explained: 


In terms of participant satisfaction, we went from, | don’t know, 48 to 60-something [of 
satisfied participations on their teacher survey]. Then, by November we were 
consistently above 80 percent satisfaction for all the rest of the sessions, including May, 
which surprised me because | thought that one wouldn’t be super popular because of 
the timing. We were able to make the shift in our process...We were able to use their 
feedback to then drive the next module to make it better. We were able to really use 
that data to make the learning stronger across the year. 


Juniper District also drew extensively on individuals and research, which together with data 
made up 84% of all information sources. There were 7 mentions of individuals and 6 mentions 
of research by Juniper District leaders as sources of information that informed their work. The 
individuals were almost all university researchers in the field of early literacy, whereas the 
research mentioned included trade books related to ELA instruction, the National Reading 
Panel meta-analysis of research related to early reading instruction, as well as three general 
invocations of “research.” Finally, there were three mentions of organizations in Juniper 
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District. Two organizations were vendors that Juniper district leaders contracted with to 
support their work with professional development related to their early literacy initiative. The 
third reference was to a nonprofit whose work informed district leaders’ efforts to design 
around phonics professional development sessions. Taken together, this suggests that while 
district leaders in all districts used similar sources of data as suggested in prior research,?” they 
differed in the degree to which they relied on some sources more than others. 


In addition to range and balance, districts also varied in the percent of district leaders that 
explicitly named any information source, which we call spread of information use. We 
considered spread to be narrow if less than 50 percent of leaders involved in decision-making 
related to ELA PD drew on information, and wide if it was greater than 50 percent. The most 
striking pattern is that three out of the four districts could be characterized as narrow spread as 
illustrated in Table 7. Information use was spread across 41.7% of district leaders involved in 
the ELA PD in Hawthorne District, 33.3% in Willow District, and 32.1% in Juniper District. This 
suggests that research and data use was concentrated in a small portion of leaders involved in 
deliberations about ELA professional development. By contrast, information use was 
widespread among Magnolia district leaders, as 85% of Magnolia district leaders referenced at 
least one source of information related to ELA professional development. Thus, in the majority 
of districts in our study, the use of research or data was found in pockets of district leaders 
involved in decision-making. 


Table 7: Spread of Information Use within Districts 


District leaders 


District leaders who 


involved in ELA PD cited information Peres 
Hawthorne District 12 5 41.7 
Willow District 12 4 33.3 
Magnolia District 15 13 86.6 
Juniper District 27 9 33.3 


By combining these three different dimensions together, the districts in our study evinced 
distinctly different portraits of information use (see Table 8). Two districts in our study - 
Hawthorne and Juniper - involved a narrow spread of district leaders who drew upon a wide 
range of information sources. While the wide range of information sources had the potential to 
bring in diverse information to inform decision-making, the information was only shared by a 
subset of those involved in deliberations. Beyond this commonality, Juniper leaders drew 
primarily on data, while Hawthorne leaders drew data, individuals, and research in roughly 
equal amount. Thus, even though both districts were faced with the same decision-making 
challenge - designing and implementing professional development in ELA - Hawthorne leaders 
relied more on external sources of information while Juniper relied more on internal sources. 


Technical Report No. 5 e 


National Center for Research in Policy and Practice 


Table 8: District portraits of information use 


Spread Range Balance 
Hawthorne District Narrow Wide Data, research, organizations 
Willow District Narrow Narrow Data, individuals, research 
Magnolia District Wide Narrow Individuals 
Juniper District Narrow Wide Data 


Willow District provided a contrast with Hawthorne and Juniper. Like those two districts, only a 
narrow spread of district leaders involved in ELA PD referenced any sources of information. 
However, unlike like those two districts, this narrow spread of leaders referenced a narrow 
range of information sources. While the narrow range of information might foster greater 
coherence, that coherence might only extend to the small percentage of individuals who drew 
on sources of information to support their work on professional development in ELA. 


Finally, Magnolia district was the only district with a wide spread of leaders involved in 
decision-making about ELA professional development who invoked information. At the same 
time, this wide range of decision makers focused on very few sources of information. This may 
suggest a greater likelihood of coherence across those involved in ELA decision-making because 
the potential for multiple ideas was reduced and shared among a substantial percentage of 
those involved in this area of work. The trend towards coherence was potentially intensified 
further because district leaders primarily identified individuals associated with their external 
partner, Education University, as the main source of information guiding their decision-making. 
Thus, not only was the range narrow, most of the information sources were related to the same 
external organization. At the same time as this particular configuration of spread, range, and 
breadth might have fostered greater coherence in decision-making, it also had the possibility of 
fostering groupthink, as there were few avenues to introduce alternative points of view or 
information into the decision-making process. 


The Relationship Between Routines and Information use 


The nature and structure of routines and routine clusters influenced what information district 
leaders drew on in their PD decision-making in all four districts. First, the type of organizational 
routine - the nature of work that was accomplished within the routine - influenced both the 
range and balance of information, accounting for variability within districts between different 
routines. Second, focusing on one source of information — research - the structure of 
connections between routines enabled research in one routine to travel to, and inform the 
decision-making of, other routines. Here, the presence or absence of connections between 
routines accounted for the variability in research use across districts. 


The role of routine type. Information use varied within districts depending on the type of 
routine. Districts in our study divided up the disparate work of decision-making into different 
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types of organizational routines. We inductively identified three types of routines: design, 
deployment, and diagnosis. Design routines involved developing a program of ELA professional 
development. Examples of design routines in Hawthorne district include the ‘Literacy Leaders 
Designing Quarterly PD routine,’ ‘Teacher Leaders Designing Quarterly PD routine,’ and 
‘Revising and Reviewing Quarterly PD routine.’ All focused on creating a PD experience for 
teachers and school coaches. Deployment routines involved district leaders delivering 
professional development to staff that they or others had designed. In Hawthorne District, an 
example of a deployment routine was the ‘Facilitating Quarterly PD routine’, where district 
leaders’ work centered on providing the PD that they previously designed. Diagnostic routines 
involved evaluating the professional development provided to teachers and other staff, 
typically using survey data from those who participated in the PD sessions. Examples of 
diagnostic routines included the ‘Evaluating Quarterly PD routine’ in Hawthorne District and the 
‘Evaluating Foundational Literacy PD routine’ in Juniper District. All four districts had design, 
deployment, and diagnosis routines. 


The type of routine influenced the range and balance of information used. First, as illustrated 
by the totals in Table 9, district leaders across all four districts described using a greater number 
of distinct information sources - what we call range - in the design routines than either 
deployment or diagnostic routines. District leaders invoked just over 70 percent of information 
sources (46 out of 65) in relation to design routines. By comparison, only 21.5% of information 
sources (14) invoked were in reference to deployment routines, and 7.7% (5) were about 
diagnostic routines. For example, in Hawthorne District, district leaders named 19 sources of 
information for their design routines, but only two sources of information for their diagnostic 
routines. They made no explicit references to information for their deployment routines. 


Table 9: Range and Balance of Information in Different Types of Routines 


Design Deployment Diagnosis 
Routines Routines Routines 
N | % N | % N | % 
Research 14 | 30.4% 1 | 7.1% 
People 14 | 30.4% 4 | 28.6% 
Organizations 5 | 10.9% 4 | 28.6% 
Data 13 | 28.3% 5 | 35.7% 5 | 100% 
Total 46 14 5 


Second, the type of routine also influenced the balance of information types invoked in 
decision-making; that is, the distribution of types of information sources. District leaders 
invoked multiple types of information in design and deployment routines, but only invoked 
data in diagnosis routines (see Table 9). The one exception to this pattern was Willow District, 
where district leaders invoked few types of information in any of their routines. They invoked 
research and an organization in design routines, organizations and data in deployment routines, 
and data in their diagnosis routine. 
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District leaders also had a greater emphasis on research in design routines than in deployment 
and diagnosis routines. Nearly a third of all information sources in design routines were 
research (14 out of 46 sources), compared to only one out of 14 data sources in deployment 
routines and no references to research in diagnosis routines. For example, in Hawthorne 
District’s Foundational Literacy routine cluster, district leaders used research to inform content 
of and approaches to presenting their PD. One district leader described how research guided 
every aspect of this design process, “We usually have research included in everything...Like 
foundational skills. There’s research to go with the foundational skills that we would be working 
on when we were talking about [classroom] routines, and station work, and that kind of thing.” 
The one exception to this pattern was Magnolia District, which invoked a single source of 
research - the Curriculum for Classroom Writing - in both the design and deployment routines. 
Indeed, this focus on a narrow range of information appears to be part of a district strategy in 
Magnolia District, as one literacy leader explained, “This year.... It’s been one article. I’ve used 
one article for TLT [routines], for Insights [routines]. [Assistant superintendents] have used it 
with principals...So, we didn’t do a whole lot with other resources. We’re like ‘we’re just gonna 
stay the course.” 


Variability in information use within different types of routines may be due to the fact that 
different types of routines involved district leaders engaging in different kinds of work. In 
design routines, for example, district leaders made decisions about the focus, format, and 
approach of professional development sessions. To this end, district leaders drew on data, such 
as student achievement data, to determine the focus of PD. They also drew on ideas from 
research, individuals, and organizations to inform their efforts to design the content of and 
their approach to PD. For example, one district leader explained: “So, if they’re designing a PD 
around phonics, they’II look at the various research out there - what do they want to draw on 
to frame the session for teachers? They'll also use research to help shape their approaches to 
different instructional strategies.” 


By contrast, in deployment routines, the nature of the work primarily focused on facilitating 
previously designed professional development sessions. As such, district leaders within 
deployment routines primarily used information that could guide their efforts to facilitate 
professional development, including the expertise of individuals and organizations external to 
the district, and school-level data that allowed district leaders to tailor professional 
development sessions to the needs of different schools. Finally, the nature of the work in the 
diagnosis routines typically focused on evaluating the implementation of previous professional 
development sessions, including its strengths and weaknesses. This may have encouraged 
district leaders to draw primarily on data, particularly survey feedback from PD participants, in 
their efforts to evaluate professional development sessions. 


Structure of routine cluster influences research use. |n this section, we focus exclusively on one 
information source: research. As we previously illustrated, district leaders rarely referenced 
research outside of design routines. However, we found that the structure of routine clusters 
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influenced the degree to which research use in design routines influenced district leaders’ work 
in other routines. More specifically, we found that the presence of input-output connections 
between individual routines enabled research from the design routines in the form of research- 
embedded artifacts to spread to and inform the work of district leaders in the deployment 
routines. When research was embedded into artifacts that in turn shaped district work, it 
constituted a hidden form of research use, which we call “latent use.” 


In design routines in all four districts, literacy leaders embedded research into artifacts such as 
professional development slide decks, facilitator guides, and other materials creating what we 
call research-embedded artifacts. For example, Juniper district leaders integrated research 
ideas from trade books, publications, academic researchers, and external partners into the slide 
deck, with the intention of guiding the focus and content of their Foundational Literacy PD. As 
one leader explained: “If they’re designing a PD around phonics, they’ll look at the various 
research out there - what do they want to draw on to frame the session for teachers? They'll 
also use research to help shape their approaches to different instructional strategies.” In one 
example of this approach, literacy leaders summarized and cited research to show how 
bilingual students develop as writers and what this means for teachers’ efforts to teach writing 
as part of a slide meant to be used in Foundational Literacy PD (See Figure 5). 


Figure 5: Example of Research-Embedded Artifact in Juniper District 


Research Slide #8 B?. 


ns i i i el a ae ae 
Bilingual students’ writing develops in ways that are different 
than those of their monolingual English speaking peers 
(Escamilla & Coady, 2001; Escamilla, 2006). For instance, code- 
switching (the alternating use of two languages) and 
interliteracy (the temporary application of linguistic elements 
from one language to the other when writing) are patterns of 
writing unique to bilingual individuals (Gort, 2006). When teachers 
model for bilingual students, they should explicitly model 
similarities and differences between the two languages to 
support their development of biliteracy. 


The degree to which research-embedded artifacts informed district leaders’ work in other 
routines depended on the structure of the routine cluster, and specifically, the presence or 
absence of input-output connections between design routines and other routines. In three of 
four districts - Hawthorne, Magnolia, and Juniper - input-output connections between routines 
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enabled research-embedded artifacts to travel and inform decision-making in multiple routines. 
For example, in Juniper District, all routines related to Foundational Literacy PD decision- 
making were connected through input-output relationships, which enabled the slide deck 
designed in a series of initial design routines to travel and inform decision-making in multiple 
PD routines (see Figure 4). More specifically, district leaders finalized the design of the PD slide 
deck in the Revising Foundational Literacy routine (labeled routine 4 in Figure 4). The output of 
this routine was the PD slide deck, which served as an input to the Training District Facilitators 
routine, where district leaders used this slide deck to guide their efforts to deliver PD. This slide 
deck was a requisite input for triggering the Training District Facilitators routine and, more 
importantly, the work of the routine revolved around this draft slide deck. In this way, 
connections between routines enabled a research-embedded slide deck that was created in the 
design routines to influence decision-making in the deployment routines, even though district 
leaders did not describe themselves as using research when delivering PD. 


There was a similar relationship between the input-output relationship, research-embedded 
artifacts, and research use in Magnolia District (see Figure 3). In this case, however, the 
research-embedded artifacts were both those designed during design routines and also the 
research-embedded curriculum (Curriculum for Classroom Writing) that the district had 
adopted. In both cases, the presence of input-output connections between the design routines 
and deployment routines meant that research embedded into artifacts informed the work of 
deployment routines, even when district leaders did not explicitly invoke research when 
discussing that work. 


In contrast, the absence of input-output connections in Willow District hindered the degree to 
which the research-embedded slide deck developed in the design routine from informed the 
work of two out of the five routines related to professional development (See Figure 2). Input- 
output connections between the Designing PD Institutes routine (labeled routine 1a in Figure 2) 
and the Facilitating PD Institutes routine (labeled routine 2a in Figure 2) within the PD Institutes 
routine cluster enabled the research-embedded PD Institute slide deck to travel and inform the 
work of this deployment routine. However, the lack of input-output connections between the 
PD Institutes routine cluster and the other routines that also supported the provision of literacy 
PD - the Designing Online PD Modules routine (labeled routine 1b in Figure 2) and the 
Delivering LSP Site-Based PD routine (labeled routine 1c in Figure 2) - meant that the work done 
to develop PD Institute slide decks in the Designing PD Institutes routine did not influence the 
work in subsequent routines. In fact, there was no use of research in the two routines not 
related to the routine cluster. 


Ultimately, we find that two dimensions of routines - the type of organizational routine and the 
nature of connections between routines - shaped patterns of information use, contributing to 
variability within and across districts. The type of organizational routine shaped the range and 
balance of information, which accounted for some of the variability we see within four 
districts. The nature of connections between routines influenced how research use in one 
routine shaped the work of district leaders in other routines. Here, the presence or absence of 
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connections between routines accounted for, in part, the variability in research use across 
districts. 


Additionally, we identify a new phenomenon: latent use of research. We argue that when 
research was embedded into artifacts that, in turn, influenced district leaders’ work, it 
constituted an additional, if often unacknowledged, way that research is used in day-to-day 
district work. Even though few district leaders in these deployment routines invoked research 
explicitly, district leaders in three out of our four districts engaged with research-embedded 
artifacts as they worked with teachers, teacher leaders, and school leaders in deployment 
routines. In this way, district leaders were using research, as embedded in slide decks and other 
artifacts, to inform their deployment of literacy PD. Because district leaders did not describe 
their work in the deployment routines as being informed by research, their engagement with 
research-embedded artifacts can be seen as a hidden form of research use. 


Technical Report No. 5 e | 39 
National Center for Research in Policy and Practice 


Discussion 


School district central offices make consequential decisions about teaching and learning every 
day that impact the opportunities and outcomes for millions of our nation’s public school 
students. As such, it is important to understand the factors that influence their instructional 
decision-making processes. In this study, we have focused on two aspects of decision-making: 
the sources of information that district leaders turn to and the role of organizational routines in 
shaping their use. We focus on the design, deployment, and diagnosis of professional 
development in English Language Arts. We find that all four districts drew on common sources 
for information, including data, research, individuals, and organizations. At the same time, 
districts varied in the number of unique information sources (range), the relative distribution of 
information types (balance), and the degree to which small or large numbers of individuals 
invoked research and other forms of information as they discussed the work (spread). As such, 
each district had distinct portraits of information use as part of their ELA professional 
development work. 


We also found that the structure of the interlocking routines through which districts 
accomplished the work of decision-making about ELA professional development influenced the 
nature of information use. Specifically, the type of the routine - whether it be design, 
deployment, or diagnosis - influenced what information district leaders used. Further, the ways 
in which individual routines were connected via input-output relationships shaped the degree 
to which research use in one routine influenced the work of district leaders in other routines. 
This suggests that understanding the architecture of district decision-making in a given realm 
may provide insight into how and why district leaders use some sources of information and not 
others, when they use information in the broader decision-making trajectory, and the ways in 
which the use of this research in one decision might influence a wide array of decisions in other 
areas. 


Our study, like all research studies, has several limitations. First, our analysis focused on just 
one area of instructional decision-making: professional development. Because we focused on a 
single area, we do not know if the findings we report here translate to other areas of 
instructional decision-making, such as selecting curricular materials, providing support for 
underperforming schools, or developing structures and strategies to support dual language 
learners, to name a few. Second, our analysis focuses on decision-making about a single school 
subject: English Language Arts. As a result, we do not know if the findings reported here would 
be similar for decision-making related to other school subjects such as mathematics or science. 
Third, our account relies on interview and document data and thus centers on what district 
leaders reported about their decision-making work. As we did not include an analysis of 
observation data, our account has limitations with respect to what district decision-makers 
actually did as they interacted with one another in practice. Fourth, and related, our account 
focuses exclusively on the ostensive aspect of organizational routines as distinct from the 
performative aspect. The ostensive is the abstract, generalized idea of the routine, whereas the 
performative is how the routine is enacted at a particular time and place.© As a result, our 
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account does not provide insights on how the performance of organizational routines shaped 
instructional decision-making and the role of research and other forms of information therein. 
This has implications for our findings, for example, on how routines coordinate instructional 
decision-making.® Our account highlights district leaders’ planned ex-ante efforts for 
coordinating decision-making, rather than more ad-hoc and informal coordination processes 
that have been identified by others who focus on the performative.° 


In spite of these limitations, our study makes a number of contributions to research and 
practice. To begin, it supports themes about from prior research on the sources of information 
that district leaders draw upon in their decision-making. Existing scholarship has found that 
district leaders draw on a range of information sources that goes much beyond research in their 
decision-making. Further, existing studies have shown that when district leaders do draw on 
research, they are more likely to invoke practitioner books than peer-reviewed journal 
articles.°* Our study confirms these findings but also goes beyond them in three main ways. 


First, we demonstrate that even though district leaders are using the same sources of 
information (e.g., research, data, organizations, and individuals), they differ in the range, 
balance, and spread of information they use. The districts in our sample varied in the range of 
sources they draw upon in their decision-making about ELA PD, from 6 information sources to 
25. While a large range of data sources may introduce multiple and diverse information into the 
decision-making process and thereby increasing access to diverse ideas, a narrow range may 
contribute to more focused and coherent deliberation. Districts also differed in the balance of 
information sources - the distribution of information across different sources. More specifically, 
we found that some districts relied predominantly on data, while others relied predominantly 
on individuals such as consultants or academics for advice. Still others drew on a more even 
balance of sources of data. Although our analysis cannot speak to the optimal balance of 
information sources, it is worth noting that a district’s own internal data provides information 
about quite different things than external research. Districts that rely on one more heavily than 
the other might be missing sources of information that might be beneficial to their decision- 
making. 


Further, not only did districts differ in the range and balance of information, they differed in the 
spread of information use - the percentage of district leaders involved in decision-making that 
explicitly invoked information. Notably, three out of four of the districts in our study had only a 
small percentage of district leaders who invoked information. This may or may not be a 
problem depending upon whether there are ways that the district leaders who are drawing 
upon research and data are able to finds ways to share it with those who are not. We found 
that one way in which district leaders were able to share research with those who did not 
invoke it was by embedding research into artifacts that were then used by district leaders 
involved in different aspects of the decision-making process; as the artifacts travelled from 
design to deployment routines, they enabled research to be shared among district leaders. In 
this way, information use by a narrow range of district leaders could inform the work of others 
involved in different aspects of decision-making process. 
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Second, we demonstrate the importance of organizational routines in shaping the use of 
research and data. Prior research has shown that organizational routines are a central medium 
through which school districts and other organizations make decisions.°° Yet, there has been 
little attention to their role in research and data use. Here, we find that organizational routines 
play a key role in variability in information use within school districts. More specifically, we 
show that district leaders draw upon different kinds of information in different types of 
routines. They were more likely to draw on various forms of research (publications, individuals, 
organizations) in design routines and various sources of data in diagnosis routines. This suggests 
that research and information use is not uniform, but likely varies in range and balance 
depending upon the nature of the work that district leaders are involved with in the routine. 
Organizational routines also contributed to variability in information use across districts. We 
show that the structure of the routine cluster - specifically, the nature of input-output 
connections - influences the degree to which research use in design routines shapes the work of 
district leaders in other routines. More specifically, we found that research use in a given 
routine was more likely to influence district leaders’ work in other routines when those routines 
were connected by input-output relationships. This finding suggests the importance of a system 
or organizational perspective on research use rather than focusing exclusively on individual 
decision-makers. Understanding how and why research is used and how it influences the work 
of district leaders making decisions about instruction may not be possible without attention to 
the ways in which organizational structures, such as the input-output relationships between 
different organizational routines, connect work in one sector of the district or one segment of 
the decision-making trajectory with work in other areas. 


Third, by analyzing how research enters into the ongoing stream of decision-making in 
interlocking routines, we show not only that research use is a process not an event, but also 
the how of that process. Research and information use is not only something that happens 
during a given meeting when decision makers gather together and make a decision. Rather, 
research and other forms of information are drawn upon at many different points during the 
design, deployment, and diagnosis process. Further, research use at one stage of the decision 
process, under the right conditions, can inform the downstream work later in the process. In 
this way, we capture the how of research use as a process stretched across different types of 
routines where the use of research (and other forms of information) shifts depending on the 
routine type and the nature of the work being undertaken therein. 


Conceptualizing research use as a process enables us to identify a new form of research use: 
latent use, in this case in the form of research-embedded artifacts. Since Carol Weiss’ 
groundbreaking studies of research use in the 1970s, scholars of evidence use in public 
agencies have focused on the same set of categories to describe different forms of research 
use: instrumental use, symbolic or political use, and conceptual use. Instrumental use refers to 
situations where research is applied to inform a specific decision, usually after weighing the 
relative costs and benefits of various options. Symbolic or political use denotes situations 
where research is applied as a political tool to influence or persuade others about a decision or 
legitimate a decision that has already been made. Conceptual use refers to situations where 
individuals change how they view a problem or possible solutions via engagement with 
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research, often outside of a specific decision.® In recent years, others have added to the list. 
The evaluation community coined the term “process use,” defined as when practitioners 
incorporate a research process as part of their decision-making.” More recent still, Carol Weiss 
and her team identified a new form of research use that emerged in the era of No Child Left 
Behind: imposed use.®°? Imposed use is when educational leaders are required to select from a 
list of “research-based” interventions or curriculum to receive federal or state funding. Based 
on our analysis, we identify and theorize another type of use which we refer to as latent 
research use. Specifically, we show that when district leaders embed research in artifacts, and 
other leaders interact with those artifacts in ways that shape their work, these leaders are ina 
sense using research when they use the artifact. Research is shaping the way these leaders do 
their work and thus influencing their decision-making. What is especially striking about this 
new form of research use is that it is quite possible that district leaders who engage with 
research embedded artifacts do not necessarily see this as a form of research use. Indeed, it 
was quite common to be the case in our study. For this reason, this form of research use is 
often hidden from view of both those engaging in latent use of research and researchers who 
study research use. 


Expanding our understanding of research use to include latent use highlights limitations in the 
ways that researchers typically study research use. By far, the main way that researchers study 
research use involves using self-report. Sometimes researchers explicitly ask educational 
leaders (via survey or interviews) about their use of research and other forms of evidence 
directly.®? Sometimes researchers, instead, ask educational leaders to explain a decision and 
code for explicit references to research and other forms of information. Indeed, the latter was 
our chief strategy. However, if researchers do not contextualize these references within the 
ongoing stream of decision-making activity in a district, they will miss the ways that research 
use in one phase of a decision-making trajectory - or research use in one routine - potentially 
influences research use in later phases or other routines if the educational leaders do not 
explicitly mention research. This suggests that we need to broaden the ways that we study 
research use to look not just at actors, but also artifacts and the way that the two interact with 
one another in making decisions. It also suggests the importance of longitudinal studies that 
have mechanisms for capturing the ways that invocations of research at one point in a decision 
trajectory shapes deliberation and action at subsequent moments where research use might be 
latent rather than explicit. 


Moving beyond scholarly studies of research use, this study also contributes to our 
understanding of the instructional decision-making process in school districts. Prior research 
has suggested that decision-making in school district central offices is complex, involving a set 
of interlocking activities that stretch across multiple units and multiple levels of the district 
central office.”° This study goes beyond these general statements to actually map the decision 
processes, in this case, related to professional development in English Language Arts. In so 
doing, we show that decision-making is not random, disorganized, or coincidental, as portrayed 
by James March’s famous and influential writing about the “garbage can” model.” Instead, 
decision-making is highly structured in patterned ways. By mapping decision-making routines, it 
is possible to see how decision-making is distributed across tasks, people, and time. It can be 
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more or less coordinated and more or less redundant depending upon how routines are 
connected to one another via input-output relationships. 


Finally, this study offers insights for practitioners and policy makers. First, our analysis suggests 
that as practitioners and policymakers work to improve whether and how research is used in 
instructional decision-making that they need to move beyond an exclusive focus on individual 
decision-makers and take the system into account. Our study shows how organizational 
arrangements shape decision-making and the role of research and other forms of information 
therein. Specifically, focusing on one aspect of the organizational structure - the organizational 
routines with and through which school district decision makers make decisions - 
fundamentally shape the decision-making process and the role of research and other forms of 
information. Hence, policymakers and practitioners eager to improve the role of research in 
decision-making should move beyond a focus on the knowledge and beliefs of individual 
decision makers and also take into account how the structure of their systems/organizations 
enable and constrain the role of research in decision-making. 


Second, our account suggest that policymakers and practitioners should think about the role of 
research and other forms of information in decision-making as not being uniform across the 
decision-making process. Instead, we urge policymakers and practitioners to take into account 
that the range and type of information might differ depending on the particular work being 
undertaken at different points across the decision-making process - design, deployment, and 
diagnosis. For instance, if policymakers and practitioners focused on the diagnosis aspect of the 
decision-making process, this may suggest that a district uses fewer and a narrower range of 
information, whereas a focus on the design aspect might suggest that a district uses more anda 
broader range of information. As a result, the challenge for practitioners and policymakers in 
working to improve the role of research in instructional decision-making requires sensitivity to 
different parts of the decision-making process and what types of information may be most 
relevant. 


Third, our work offers a practical design tool - routine maps - that may be useful for 
practitioners to map out the array of different decision processes within their systems, and the 
role of research therein. Routine maps may support practitioners in visualizing recurring 
patterns of decision-making activity that might otherwise be invisible and taken-for-granted. By 
mapping out decision-making routines, practitioners can examine, interrogate, and possibly 
reconfigure, how decision-making occurs, who is involved, and the role of research in the 
process. 
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